The size of video data is normally large. For this reason, devices handling video data normally encode and thereby compress the video data before transmitting the video data to a different device or storing the video data in a storage device. Widely used video coding schemes are Moving Picture Experts Group phase 2 (MPEG-2), MPEG-4, and H.264 MPEG-4 Advanced Video Coding (H.264 MPEG-4 AVC) standardized by the International Organization for Standardization/International Electrotechnical Commission (ISO/IEC). In addition, High Efficiency Video Coding (HEVC, MPEG-H/H.265) is standardized as a new coding standard.
Consideration has been made to use, in such a coding scheme, the rate distortion optimized quantization (RDOQ) for the purpose of optimizing the balance between the quality of video obtained by decoding encoded video data and the compression efficiency (refer to, for example, Japanese Laid-open Patent Publication No. 2011-176483, Published Japanese Translation of PCT International Publication for Patent Application (Kohyo) No. 2011-514055, and Japanese Laid-open Patent Publication No. 2013-110740).
In RDOQ, an orthogonal transform coefficient, which is obtained by orthogonally transforming a prediction error signal of each pixel of each block obtained by dividing each picture included in video data, is quantized in such a way so as to determine a quantized coefficient possible to minimize the coding cost taking into account both a coding error and an amount of coding.